iT邦幫忙

2022 iThome 鐵人賽

DAY 19
0
DevOps

5哥窟-5G與肝臟買賣系列 第 19

19-Istio free5gc 核心網路跨叢集部署

  • 分享至 

  • xImage
  •  

測試環境版本:
> istio: 1.13.1
> kubernetes: v1.23.4

本次測試使用開源專案free5gmano提供的free5gc 3.0.6部署yaml進行修改
free5gmano連結

*** 部署free5gc前請先安裝好gtp5g ***

git clone https://github.com/PrinzOwO/gtp5g.git
cd gtp5g
make clean && make
make install

驗證安裝結果

lsmod | grep gtp

本篇會將nrf部署至remote cluster來進行跨叢集連通測試
部署示意圖

部署yaml檔

01-free5gc-mongodb.yaml

apiVersion: v1
kind: Service
metadata:
  name: mongodb-svc
  labels: 
    service: mongodb
spec:
  ports:
  - port: 27017
  selector:
    app: free5gc-mongodb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-mongodb
spec:
  selector:
    matchLabels:
      app: free5gc-mongodb
  template:
    metadata:
      labels:
        app: free5gc-mongodb
    spec:
      containers:
      - image: free5gmano/nextepc-mongodb
        name: free5gc-mongodb
        ports:
        - containerPort: 27017
          name: mongodb
        volumeMounts:
        - name: mongodb-persistent-storage
          mountPath: /data/db
      volumes:
      - name: mongodb-persistent-storage
        persistentVolumeClaim:
          claimName: mongodb-pv-claim
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mongodb-pv-volume
  labels:
    type: local
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

02-free5gc-upf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-upf-configmap
data:
  upfcfg.yaml: |
    info:
      version: 1.0.0
      description: UPF configuration

    configuration:
      # the kind of log output
      # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
      # ReportCaller: enable the caller report or not, value: true or false
      debugLevel: debug
      ReportCaller: true

      # The IP list of the N4 interface on this UPF (Can't set to 0.0.0.0)
      pfcp:
        - addr: free5gc-upf-svc

      # The IP list of the N3/N9 interfaces on this UPF
      # If there are multiple connection, set addr to 0.0.0.0 or list all the addresses
      gtpu:
        - addr: free5gc-upf-svc
        # [optional] gtpu.name
        # - name: upf.5gc.nctu.me
        # [optional] gtpu.ifname
        # - ifname: gtpif

      # The DNN list supported by UPF
      dnn_list:
        - dnn: internet                       # Data Network Name
          cidr: 60.60.0.0/24                  # Classless Inter-Domain Routing for assigned IPv4 pool of UE
          # [optional] dnn_list[*].natifname
          # natifname: eth0
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-upf-svc
spec:
  selector:
    app: free5gc-upf
  clusterIP: None # 10.96.100.10
  ports:
    - name: sbi
      port: 29518
      protocol: TCP
    - name: gtp
      protocol: UDP
      port: 2152
    - name: pfcp
      protocol: UDP
      port: 8805
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-upf-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-upf
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-upf
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
      - name: free5gc-upf-container
        image: free5gmano/free5gc-user-plane:stage3.0.6-amd64
        securityContext:
          privileged: true
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && /setup-uptun.sh eth0 && ./bin/free5gc-upfd"]
        volumeMounts:
        - mountPath: /dev/net/tun
          name: tun-volume
        - name: upfcfg
          mountPath: /free5gc/NFs/upf/build/config/upfcfg.yaml
          subPath: upfcfg.yaml
      volumes:
      - name: tun-volume
        hostPath:
          path: /dev/net/tun
      - name: upfcfg
        configMap:
          name: free5gc-upf-configmap
          items:
          - key: upfcfg.yaml
            path: upfcfg.yaml

03-free5gc-nrf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-nrf-configmap
  labels:
    configmap: free5gc-nrf
data:
  nrfcfg.yaml: |
    info:
      version: 1.0.0
      description: NRF initial local configuration

    configuration:
      MongoDBName: free5gc                    # database name in MongoDB
      MongoDBUrl: mongodb://mongodb-svc.5gc:27017 # a valid URL of the mongodb
      DefaultServiceIP: "free5gc-nrf-svc"
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-nrf-svc         # IP used to serve NFs or register to another NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29510                           # port used to bind the service
      DefaultPlmnId:
        mcc: 208                              # Mobile Country Code (3 digits string, digit: 0~9)
        mnc: 93                               # Mobile Network Code (2 or 3 digits string, digit: 0~9)
      serviceNameList:                        # the SBI services provided by this NRF, refer to TS 29.510
        - nnrf-nfm                            # Nnrf_NFManagement service
        - nnrf-disc                           # Nnrf_NFDiscovery service

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      NRF:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
      MongoDBLibrary:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-nrf-svc
  labels:
    app: free5gc-nrf
    service: free5gc-nrf
spec:
  selector:
    app: free5gc-nrf
  ports:
    - name: sbi
      protocol: TCP
      port: 29510
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-nrf-deployment-v1
  labels: 
    app: free5gc-nrf
    version: v1
spec:
  selector:
    matchLabels:
      app: free5gc-nrf
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-nrf
        version: v1
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/nrf"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: nrfcfg
          mountPath: /free5gc/config/nrfcfg.yaml
          subPath: nrfcfg.yaml
      volumes:
      - name: nrfcfg
        configMap:
          name: free5gc-nrf-configmap
          items:
          - key: nrfcfg.yaml
            path: nrfcfg.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-nrf-deployment-v2
  labels: 
    app: free5gc-nrf
    version: v2
spec:
  selector:
    matchLabels:
      app: free5gc-nrf
      version: v2
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-nrf
        version: v2
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 300s && ./bin/nrf"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: nrfcfg
          mountPath: /free5gc/config/nrfcfg.yaml
          subPath: nrfcfg.yaml
      volumes:
      - name: nrfcfg
        configMap:
          name: free5gc-nrf-configmap
          items:
          - key: nrfcfg.yaml
            path: nrfcfg.yaml

04-free5gc-amf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-amf-configmap
data:
  amfcfg.yaml: |
    info:
      version: 1.0.2
      description: AMF initial local configuration

    configuration:
      amfName: AMF                            # the name of this AMF
      ngapIpList:                             # the IP list of N2 interfaces on this AMF
        - 0.0.0.0
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-amf-svc         # IP used to register to NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29518                           # port used to bind the service
      serviceNameList:                        # the SBI services provided by this AMF, refer to TS 29.518
        - namf-comm                           # Namf_Communication service
        - namf-evts                           # Namf_EventExposure service
        - namf-mt                             # Namf_MT service
        - namf-loc                            # Namf_Location service
        - namf-oam                            # OAM service
      servedGuamiList:                        # Guami (Globally Unique AMF ID) list supported by this AMF
      # <GUAMI> = <MCC><MNC><AMF ID>
        - plmnId:                             # Public Land Mobile Network ID, <PLMN ID> = <MCC><MNC>
            mcc: 208                          # Mobile Country Code (3 digits string, digit: 0~9)
            mnc: 93                           # Mobile Network Code (2 or 3 digits string, digit: 0~9)
          amfId: cafe00                       # AMF identifier (3 bytes hex string, range: 000000~FFFFFF)
      supportTaiList:                         # the TAI (Tracking Area Identifier) list supported by this AMF
        - plmnId:
            mcc: 208
            mnc: 93
          tac: 1                              # Tracking Area Code (uinteger, range: 0~16777215)
      plmnSupportList:                        # the PLMNs (Public land mobile network) list supported by this AMF
        - plmnId:
            mcc: 208
            mnc: 93
          snssaiList:                         # the S-NSSAI (Single Network Slice Selection Assistance Information) list supported by this AMF
            - sst: 1
              sd: 010203
            - sst: 1
              sd: 112233
      supportDnnList:                         # the DNN (Data Network Name) list supported by this AMF
        - internet
      nrfUri: http://free5gc-nrf-svc.5gc:29510    # a valid URI of NRF
      security:                               # NAS security parameters
        integrityOrder:                       # the priority of integrity algorithms
          - NIA2
          - NIA0
        cipheringOrder:                       # the priority of ciphering algorithms
          - NEA0
          - NEA2
      networkName:                            # the name of this core network
        full: free5GC
        short: free
      t3502Value: 720                         # timer value (seconds) at UE side
      t3512Value: 3600                        # timer value (seconds) at UE side
      non3gppDeregistrationTimerValue: 3240   # timer value (seconds) at UE side
      # retransmission timer for paging message
      t3513:
        enable: true                          # true or false
        expireTime: 6s                        # default is 6 seconds
        maxRetryTimes: 4                      # the max number of retransmission
      # retransmission timer for NAS Deregistration Request message
      t3522:
        enable: true
        expireTime: 6s
        maxRetryTimes: 4
      # retransmission timer for NAS Registration Accept message
      t3550:
        enable: true
        expireTime: 6s
        maxRetryTimes: 4
      # retransmission timer for NAS Authentication Request/Security Mode Command message
      t3560:
        enable: true
        expireTime: 6s
        maxRetryTimes: 4
      # retransmission timer for NAS Notification message
      t3565:
        enable: true
        expireTime: 6s
        maxRetryTimes: 4

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      AMF:
        debugLevel: debug
        ReportCaller: true
      NAS:
        debugLevel: debug
        ReportCaller: true
      FSM:
        debugLevel: debug
        ReportCaller: true
      NGAP:
        debugLevel: debug
        ReportCaller: true
      Aper:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-amf-svc
spec:
  selector:
    app: free5gc-amf
  ports:
    - name: sbi
      protocol: TCP
      port: 29518
    - name: sctp
      protocol: SCTP
      port: 38412
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-amf-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-amf
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-amf
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/amf"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: amfcfg
          mountPath: /free5gc/config/amfcfg.yaml
          subPath: amfcfg.yaml
      volumes:
      - name: amfcfg
        configMap:
          name: free5gc-amf-configmap
          items: 
          - key: amfcfg.yaml
            path: amfcfg.yaml

05-free5gc-smf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-smf-configmap
data:
  smfcfg.yaml: |
    info:
      version: 1.0.2
      description: SMF initial local configuration

    configuration:
      smfName: SMF                            # the name of this SMF
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-smf-svc         # IP used to register to NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29502                           # Port used to bind the service
        tls:                                  # the local path of TLS key
          key: free5gc/support/TLS/smf.key    # SMF TLS Certificate
          pem: free5gc/support/TLS/smf.pem    # SMF TLS Private key
      serviceNameList:                        # the SBI services provided by this SMF, refer to TS 29.502
        - nsmf-pdusession                     # Nsmf_PDUSession service
        - nsmf-event-exposure                 # Nsmf_EventExposure service
        - nsmf-oam                            # OAM service
      snssaiInfos:                            # the S-NSSAI (Single Network Slice Selection Assistance Information) list supported by this AMF
        - sNssai:                             # S-NSSAI (Single Network Slice Selection Assistance Information)
            sst: 1                            # Slice/Service Type (uinteger, range: 0~255)
            sd: 010203                        # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
          dnnInfos:                           # DNN information list
            - dnn: internet                   # Data Network Name
              dns:                            # the IP address of DNS
                ipv4: 8.8.8.8
        - sNssai:                             # S-NSSAI (Single Network Slice Selection Assistance Information)
            sst: 1                            # Slice/Service Type (uinteger, range: 0~255)
            sd: 112233                        # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
          dnnInfos:                           # DNN information list
            - dnn: internet                   # Data Network Name
              dns:                            # the IP address of DNS
                ipv4: 8.8.8.8
      pfcp:                                   # the IP address of N4 interface on this SMF (PFCP)
        addr: free5gc-smf-svc
      userplane_information:                  # list of userplane information
        up_nodes:                             # information of userplane node (AN or UPF)
          gNB1:                               # the name of the node
            type: AN                          # the type of the node (AN or UPF)
          UPF:                                # the name of the node
            type: UPF                         # the type of the node (AN or UPF)
            node_id: free5gc-upf-svc          # the IP/FQDN of N4 interface on this UPF (PFCP)
            sNssaiUpfInfos:                   # S-NSSAI information list for this UPF
              - sNssai:                       # S-NSSAI (Single Network Slice Selection Assistance Information)
                  sst: 1                      # Slice/Service Type (uinteger, range: 0~255)
                  sd: 010203                  # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
                dnnUpfInfoList:               # DNN information list for this S-NSSAI
                  - dnn: internet
                    pools:
                      - cidr: 60.60.0.0/16
              - sNssai:                       # S-NSSAI (Single Network Slice Selection Assistance Information)
                  sst: 1                      # Slice/Service Type (uinteger, range: 0~255)
                  sd: 112233                  # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
                dnnUpfInfoList:               # DNN information list for this S-NSSAI
                  - dnn: internet
                    pools:
                      - cidr: 60.61.0.0/16
            interfaces:                       # Interface list for this UPF
              - interfaceType: N3             # the type of the interface (N3 or N9)
                endpoints:                    # the IP address of this N3/N9 interface on this UPF
                  - free5gc-upf-svc
                networkInstance: internet     # Data Network Name (DNN)
        links: # the topology graph of userplane, A and B represent the two nodes of each link
          - A: gNB1
            B: UPF
      nrfUri: http://free5gc-nrf-svc.5gc:29510    # a valid URI of NRF
      locality: area1                         # Name of the location where a set of AMF, SMF and UPFs are located

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      SMF:
        debugLevel: debug
        ReportCaller: true
      NAS:
        debugLevel: debug
        ReportCaller: true
      NGAP:
        debugLevel: debug
        ReportCaller: true
      Aper:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
      PFCP:
        debugLevel: debug
        ReportCaller: true

  uerouting.yaml: |
    info:
      version: 1.0.1
      description: Routing information for UE

    ueRoutingInfo:                            # the list of UE routing information
      UE1:                                    # Group Name
        members:
        - imsi-2089300000003                  # Subscription Permanent Identifier of the UE
        topology:                             # Network topology for this group (Uplink: A->B, Downlink: B->A)
        # default path derived from this topology
        # node name should be consistent with smfcfg.yaml
          - A: gNB1
            B: BranchingUPF
          - A: BranchingUPF
            B: AnchorUPF1
        specificPath:
          - dest: 60.60.0.103/32 # the destination IP address on Data Network (DN)
            # the order of UPF nodes in this path. We use the UPF's name to represent each UPF node.
            # The UPF's name should be consistent with smfcfg.yaml
            path: [BranchingUPF, AnchorUPF2]

      UE2: # Group Name
        members:
        - imsi-2089300000004 # Subscription Permanent Identifier of the UE
        topology: # Network topology for this group (Uplink: A->B, Downlink: B->A)
        # default path derived from this topology
        # node name should be consistent with smfcfg.yaml
          - A: gNB1
            B: BranchingUPF
          - A: BranchingUPF
            B: AnchorUPF1
        specificPath:
          - dest: 10.0.0.11/32 # the destination IP address on Data Network (DN)
            # the order of UPF nodes in this path. We use the UPF's name to represent each UPF node.
            # The UPF's name should be consistent with smfcfg.yaml
            path: [BranchingUPF, AnchorUPF2]
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-smf-svc
spec:
  selector:
    app: free5gc-smf
  clusterIP: None # 10.96.100.3
  ports:
    - name: sbi
      protocol: TCP
      port: 29502
    - name: pfcp
      protocol: UDP
      port: 8805
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-smf-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-smf
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-smf
      # annotations:
      #   sidecar.istio.io/inject: "false"
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/smf"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: smfcfg
          mountPath: free5gc/config/smfcfg.yaml
          subPath: smfcfg.yaml
        - name: uerouting
          mountPath: free5gc/config/uerouting.yaml
          subPath: uerouting.yaml
      volumes:
      - name: smfcfg
        configMap:
          name: free5gc-smf-configmap
          items: 
          - key: smfcfg.yaml
            path: smfcfg.yaml
      - name: uerouting
        configMap:
          name: free5gc-smf-configmap
          items: 
          - key: uerouting.yaml
            path: uerouting.yaml

06-free5gc-udr.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-udr-configmap
data:
  udrcfg.yaml: |
    info:
      version: 1.0.0
      description: UDR initial local configuration

    configuration:
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-udr-svc         # IP used to register to NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29504                           # port used to bind the service
      mongodb:
        name: free5gc                         # Database name in MongoDB
        url: mongodb://mongodb-svc.5gc:27017      # URL of MongoDB
      nrfUri: http://free5gc-nrf-svc.5gc:29510    # a valid URI of NRF

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      UDR:
        debugLevel: debug
        ReportCaller: true
      MongoDBLibrary:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-udr-svc
spec:
  selector:
    app: free5gc-udr
  # clusterIP: None # 10.96.100.8
  ports:
    - name: sbi
      protocol: TCP
      port: 29504
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-udr-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-udr
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-udr
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/udr"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: udrcfg
          mountPath: /free5gc/config/udrcfg.yaml
          subPath: udrcfg.yaml
      volumes:
      - name: udrcfg
        configMap:
          name: free5gc-udr-configmap
          items:
          - key: udrcfg.yaml
            path: udrcfg.yaml

07-free5gc-pcf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-pcf-configmap
data:
  pcfcfg.yaml: |
    info:
      version: 1.0.0
      description: PCF initial local configuration

    configuration:
      pcfName: PCF                                # the name of this PCF
      sbi:                                        # Service-based interface information
        scheme: http                              # the protocol for sbi (http or https)
        registerIPv4: free5gc-pcf-svc             # IP used to register to NRF
        bindingIPv4: 0.0.0.0                      # IP used to bind the service
        port: 29507                               # port used to bind the service
      timeFormat: 2019-01-02 15:04:05             # time format of this PCF
      defaultBdtRefId: BdtPolicyId-               # BDT Reference ID, indicating transfer policies of background data transfer.
      nrfUri: http://free5gc-nrf-svc.5gc:29510        # a valid URI of NRF
      serviceList:   # the SBI services provided by this PCF, refer to TS 29.507
        - serviceName: npcf-am-policy-control     # Npcf_AMPolicyControl service
        - serviceName: npcf-smpolicycontrol       # Npcf_SMPolicyControl service
          suppFeat: 3fff # the features supported by Npcf_SMPolicyControl, name defined in TS 29.512 5.8-1, value defined in TS 29.571 5.2.2
        - serviceName: npcf-bdtpolicycontrol      # Npcf_BDTPolicyControl service
        - serviceName: npcf-policyauthorization   # Npcf_PolicyAuthorization service
          suppFeat: 3    # the features supported by Npcf_PolicyAuthorization, name defined in TS 29.514 5.8-1, value defined in TS 29.571 5.2.2
        - serviceName: npcf-eventexposure         # Npcf_EventExposure service
        - serviceName: npcf-ue-policy-control     # Npcf_UEPolicyControl service
      mongodb:                                    # the mongodb connected by this PCF
        name: free5gc                             # name of the mongodb
        url: mongodb://mongodb-svc.5gc:27017          # a valid URL of the mongodb

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      PCF:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-pcf-svc
spec:
  selector:
    app: free5gc-pcf
  # clusterIP: None # 10.96.100.9
  ports:
    - name: sbi
      protocol: TCP
      port: 29507
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-pcf-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-pcf
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-pcf
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/pcf"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: pcfcfg
          mountPath: /free5gc/config/pcfcfg.yaml
          subPath: pcfcfg.yaml
      volumes:
      - name: pcfcfg
        configMap:
          name: free5gc-pcf-configmap
          items:
          - key: pcfcfg.yaml
            path: pcfcfg.yaml

08-free5gc-udm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-udm-configmap
data:
  udmcfg.yaml: |
    info:
      version: 1.0.1
      description: UDM initial local configuration

    configuration:
      serviceNameList:                        # the SBI services provided by this UDM, refer to TS 29.503
        - nudm-sdm                            # Nudm_SubscriberDataManagement service
        - nudm-uecm                           # Nudm_UEContextManagement service
        - nudm-ueau                           # Nudm_UEAuthenticationManagement service
        - nudm-ee                             # Nudm_EventExposureManagement service
        - nudm-pp                             # Nudm_ParameterProvisionDataManagement service
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-udm-svc         # IP used to register to NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29503                           # Port used to bind the service
        tls:                                  # the local path of TLS key
          log: free5gc/udmsslkey.log          # UDM keylog
          pem: free5gc/support/TLS/udm.pem    # UDM TLS Certificate
          key: free5gc/support/TLS/udm.key    # UDM TLS Private key
      nrfUri: http://free5gc-nrf-svc.5gc:29510    # a valid URI of NRF

      # test data set from TS33501-f60 Annex C.4
      SuciProfile:
        - ProtectionScheme: 1                 # Protect Scheme: Profile A
          PrivateKey: c53c22208b61860b06c62e5406a7b330c2b577aa5558981510d128247d38bd1d
          PublicKey: 5a8d38864820197c3394b92613b20b91633cbd897119273bf8e4a6f4eec0a650
        - ProtectionScheme: 2                 # Protect Scheme: Profile B
          PrivateKey: F1AB1074477EBCC7F554EA1C5FC368B1616730155E0041AC447D6301975FECDA
          PublicKey: 0472DA71976234CE833A6907425867B82E074D44EF907DFB4B3E21C1C2256EBCD15A7DED52FCBB097A4ED250E036C7B9C8C7004C4EEDC4F068CD7BF8D3F900E3B4

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      UDM:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-udm-svc
spec:
  selector:
    app: free5gc-udm
  # clusterIP: None # 10.96.100.7
  ports:
    - name: sbi
      protocol: TCP
      port: 29503
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-udm-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-udm
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-udm
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/udm"]
        volumeMounts:
        - name: udmcfg
          mountPath: /free5gc/config/udmcfg.yaml
          subPath: udmcfg.yaml
      volumes:
      - name: udmcfg
        configMap:
          name: free5gc-udm-configmap
          items:
          - key: udmcfg.yaml
            path: udmcfg.yaml

09-free5gc-nssf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-nssf-configmap
data:
  nssfcfg.yaml: |
    info:
      version: 1.0.0
      description: NSSF initial local configuration

    configuration:
      nssfName: NSSF                          # the name of this NSSF
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-nssf-svc        # IP used to register to NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29531                           # Port used to bind the service
      serviceNameList:                        # the SBI services provided by this SMF, refer to TS 29.531
        - nnssf-nsselection                   # Nnssf_NSSelection service
        - nnssf-nssaiavailability             # Nnssf_NSSAIAvailability service
      nrfUri: http://free5gc-nrf-svc.5gc:29510    # a valid URI of NRF
      supportedPlmnList:                      # the PLMNs (Public land mobile network) list supported by this NSSF
        - mcc: 208                            # Mobile Country Code (3 digits string, digit: 0~9)
          mnc: 93                             # Mobile Network Code (2 or 3 digits string, digit: 0~9)
      supportedNssaiInPlmnList:               # Supported S-NSSAI List for each PLMN
        - plmnId:                             # Public Land Mobile Network ID, <PLMN ID> = <MCC><MNC>
            mcc: 208                          # Mobile Country Code (3 digits string, digit: 0~9)
            mnc: 93                           # Mobile Network Code (2 or 3 digits string, digit: 0~9)
          supportedSnssaiList:                # Supported S-NSSAIs of the PLMN
            - sst: 1                          # Slice/Service Type (uinteger, range: 0~255)
              sd: 010203                      # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
            - sst: 1
              sd: 112233
            - sst: 1
              sd: 000003
            - sst: 2
              sd: 000001
            - sst: 2
              sd: 000002
      nsiList:                                # List of available Network Slice Instance (NSI)
        - snssai:                             # S-NSSAI of this NSI
            sst: 1                            # Slice/Service Type (uinteger, range: 0~255)
          nsiInformationList:                 # Information list of this NSI
            # the NRF to be used to select the NFs/services within the selected NSI, and an optional ID
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 10
        - snssai:                             # S-NSSAI of this NSI
            sst: 1                            # Slice/Service Type (uinteger, range: 0~255)
            sd: 000001                        # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
          nsiInformationList:                 # Information list of this NSI
            # the NRF to be used to select the NFs/services within the selected NSI, and an optional ID
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 11
        - snssai:
            sst: 1
            sd: 000002
          nsiInformationList:
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 12
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 12
        - snssai:
            sst: 1
            sd: 000003
          nsiInformationList:
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 13
        - snssai:
            sst: 2
          nsiInformationList:
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 20
        - snssai:
            sst: 2
            sd: 000001
          nsiInformationList:
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 21
        - snssai:
            sst: 1
            sd: 010203
          nsiInformationList:
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 22
        - snssai:
            sst: 1
            sd: 112233
          nsiInformationList:
            - nrfId: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
              nsiId: 23
      amfSetList:                             # List of AMF Sets that my be assigned by this NSSF
        - amfSetId: 1                         # the AMF Set identifier
          amfList:                            # Instance ID of the AMFs in this set
            - ffa2e8d7-3275-49c7-8631-6af1df1d9d26
            - 0e8831c3-6286-4689-ab27-1e2161e15cb1
            - a1fba9ba-2e39-4e22-9c74-f749da571d0d
          # URI of the NRF used to determine the list of candidate AMF(s) from the AMF Set
          nrfAmfSet: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
          # the Nssai availability data information per TA supported by the AMF
          supportedNssaiAvailabilityData:
            - tai:                            # Tracking Area Identifier
                plmnId:                       # Public Land Mobile Network ID, <PLMN ID> = <MCC><MNC>
                  mcc: 466                    # Mobile Country Code (3 digits string, digit: 0~9)
                  mnc: 92                     # Mobile Network Code (2 or 3 digits string, digit: 0~9)
                tac: 33456                    # Tracking Area Code (uinteger, range: 0~16777215)
              supportedSnssaiList:            # Supported S-NSSAIs of the tracking area
                - sst: 1                      # Slice/Service Type (uinteger, range: 0~255)
                  sd: 000001                  # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
                - sst: 1
                  sd: 000002
                - sst: 2
                  sd: 000001
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33457
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 1
                  sd: 000002
        - amfSetId: 2                         # the AMF Set identifier
          nrfAmfSet: http://free5gc-nrf-svc.5gc:29510/nnrf-nfm/v1/nf-instances
          supportedNssaiAvailabilityData:
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33456
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 1
                  sd: 000003
                - sst: 2
                  sd: 000001
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33458
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 2
      amfList:                                          # List of AMFs that may be assigned by this NSSF
        - nfId: 469de254-2fe5-4ca0-8381-af3f500af77c    # ID of this AMF
          # The NSSAI availability data information per TA supported by the AMF
          supportedNssaiAvailabilityData:
            - tai:                                      # Tracking Area Identifier
                plmnId:                                 # Public Land Mobile Network ID, <PLMN ID> = <MCC><MNC>
                  mcc: 466                              # Mobile Country Code (3 digits string, digit: 0~9)
                  mnc: 92                               # Mobile Network Code (2 or 3 digits string, digit: 0~9)
                tac: 33456                              # Tracking Area Code (uinteger, range: 0~16777215)
              supportedSnssaiList:                      # Supported S-NSSAIs of the tracking area
                - sst: 1                                # Slice/Service Type (uinteger, range: 0~255)
                - sst: 1                                # Slice/Service Type (uinteger, range: 0~255)
                  sd: 000002                            # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
                - sst: 2                                # Slice/Service Type (uinteger, range: 0~255)
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33457
              supportedSnssaiList:
                - sst: 1
                  sd: 000001
                - sst: 1
                  sd: 000002
        - nfId: fbe604a8-27b2-417e-bd7c-8a7be2691f8d    # ID of this AMF
          supportedNssaiAvailabilityData:
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33458
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 1
                  sd: 000003
                - sst: 2
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33459
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 2
                - sst: 2
                  sd: 000001
        - nfId: b9e6e2cb-5ce8-4cb6-9173-a266dd9a2f0c    # ID of this AMF
          supportedNssaiAvailabilityData:
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33456
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 1
                  sd: 000002
                - sst: 2
            - tai:
                plmnId:
                  mcc: 466
                  mnc: 92
                tac: 33458
              supportedSnssaiList:
                - sst: 1
                - sst: 1
                  sd: 000001
                - sst: 2
                - sst: 2
                  sd: 000001
      taList:                                 # List of supported tracking area and their related information of this NSSF instance
        - tai:                                # Tracking Area Identity
            plmnId:                           # Public Land Mobile Network ID, <PLMN ID> = <MCC><MNC>
              mcc: 466                        # Mobile Country Code (3 digits string, digit: 0~9)
              mnc: 92                         # Mobile Network Code (2 or 3 digits string, digit: 0~9)
            tac: 33456                        # Tracking Area Code (uinteger, range: 0~16777215)
          accessType: 3GPP_ACCESS             # Access type of the tracking area
          supportedSnssaiList:                # List of supported S-NSSAIs of the tracking area
            - sst: 1                          # Slice/Service Type (uinteger, range: 0~255)
            - sst: 1                          # Slice/Service Type (uinteger, range: 0~255)
              sd: 000001                      # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
            - sst: 1
              sd: 000002
            - sst: 2
        - tai:
            plmnId:
              mcc: 466
              mnc: 92
            tac: 33457
          accessType: 3GPP_ACCESS
          supportedSnssaiList:
            - sst: 1
            - sst: 1
              sd: 000001
            - sst: 1
              sd: 000002
            - sst: 2
        - tai:
            plmnId:
              mcc: 466
              mnc: 92
            tac: 33458
          accessType: 3GPP_ACCESS
          supportedSnssaiList:
            - sst: 1
            - sst: 1
              sd: 000001
            - sst: 1
              sd: 000003
            - sst: 2
          restrictedSnssaiList:
            - homePlmnId:
                mcc: 310
                mnc: 560
              sNssaiList:
                - sst: 1
                  sd: 000003
        - tai:
            plmnId:
              mcc: 466
              mnc: 92
            tac: 33459
          accessType: 3GPP_ACCESS
          supportedSnssaiList:
            - sst: 1
            - sst: 1
              sd: 000001
            - sst: 2
            - sst: 2
              sd: 000001
          restrictedSnssaiList:
            - homePlmnId:
                mcc: 310
                mnc: 560
              sNssaiList:
                - sst: 2
                  sd: 000001
      # List of mappings of S-NSSAI in the serving network and the value of the home network
      mappingListFromPlmn:
        - operatorName: NTT Docomo            # Home PLMN name
          homePlmnId:                         # Home PLMN identifier
            mcc: 440                          # Mobile Country Code (3 digits string, digit: 0~9)
            mnc: 10                           # Mobile Network Code (2 or 3 digits string, digit: 0~9)
          mappingOfSnssai:                    # List of S-NSSAIs mapping
            - servingSnssai:                  # S-NSSAI in the serving network
                sst: 1                        # Slice/Service Type (uinteger, range: 0~255)
                sd: 000001                    # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
              homeSnssai:                     # S-NSSAI in the home network
                sst: 1                        # Slice/Service Type (uinteger, range: 0~255)
                sd: 1                         # Slice Differentiator (3 bytes hex string, range: 000000~FFFFFF)
            - servingSnssai:
                sst: 1
                sd: 000002
              homeSnssai:
                sst: 1
                sd: 000003
            - servingSnssai:
                sst: 1
                sd: 000003
              homeSnssai:
                sst: 1
                sd: 000004
            - servingSnssai:
                sst: 2
                sd: 000001
              homeSnssai:
                sst: 2
                sd: 000002
        - operatorName: AT&T Mobility         # Home PLMN name
          homePlmnId:
            mcc: 310
            mnc: 560
          mappingOfSnssai:
            - servingSnssai:
                sst: 1
                sd: 000001
              homeSnssai:
                sst: 1
                sd: 000002
            - servingSnssai:
                sst: 1
                sd: 000002
              homeSnssai:
                sst: 1
                sd: 000003

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      NSSF:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-nssf-svc
spec:
  selector:
    app: free5gc-nssf
  # clusterIP: None # 10.96.100.6
  ports:
    - name: sbi
      protocol: TCP
      port: 29531
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-nssf-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-nssf
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-nssf
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/nssf"]
        env:
        - name: GIN_MODE
          value: release # debug | release
        volumeMounts:
        - name: nssfcfg
          mountPath: /free5gc/config/nssfcfg.yaml
          subPath: nssfcfg.yaml
      volumes:
      - name: nssfcfg
        configMap:
          name: free5gc-nssf-configmap
          items:
          - key: nssfcfg.yaml
            path: nssfcfg.yaml

10-free5gc-ausf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: free5gc-ausf-configmap
data:
  ausfcfg.yaml: |
    info:
      version: 1.0.0
      description: AUSF initial local configuration

    configuration:
      sbi:                                    # Service-based interface information
        scheme: http                          # the protocol for sbi (http or https)
        registerIPv4: free5gc-ausf-svc        # IP used to register to NRF
        bindingIPv4: 0.0.0.0                  # IP used to bind the service
        port: 29509                           # Port used to bind the service
      serviceNameList:                        # the SBI services provided by this AUSF, refer to TS 29.509
        - nausf-auth                          # Nausf_UEAuthentication service
      nrfUri: http://free5gc-nrf-svc.5gc:29510    # a valid URI of NRF
      plmnSupportList:                        # the PLMNs (Public Land Mobile Network) list supported by this AUSF
        - mcc: 208                            # Mobile Country Code (3 digits string, digit: 0~9)
          mnc: 93                             # Mobile Network Code (2 or 3 digits string, digit: 0~9)
        - mcc: 123
          mnc: 45
      groupId: ausfGroup001                   # ID for the group of the AUSF

    # the kind of log output
    # debugLevel: how detailed to output, value: trace, debug, info, warn, error, fatal, panic
    # ReportCaller: enable the caller report or not, value: true or false
    logger:
      AUSF:
        debugLevel: debug
        ReportCaller: true
      PathUtil:
        debugLevel: debug
        ReportCaller: true
      OpenApi:
        debugLevel: debug
        ReportCaller: true
---
apiVersion: v1
kind: Service
metadata:
  name: free5gc-ausf-svc
spec:
  selector:
    app: free5gc-ausf
  # clusterIP: None # 10.96.100.4
  ports:
    - name: sbi
      protocol: TCP
      port: 29509
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: free5gc-ausf-deployment
spec:
  selector:
    matchLabels:
      app: free5gc-ausf
  replicas: 1
  template:
    metadata:
      labels:
        app: free5gc-ausf
    spec:
      containers:
      - name: myapp-container
        image: free5gmano/free5gc-control-plane:stage3.0.6-amd64
        command: ["/bin/sh"]
        args: ["-c", "sleep 5s && ./bin/ausf"]
        env:
        - name: GIN_MODE
          value: debug # debug | release
        volumeMounts:
        - name: ausfcfg
          mountPath: /free5gc/config/ausfcfg.yaml
          subPath: ausfcfg.yaml
      volumes:
      - name: ausfcfg
        configMap:
          name: free5gc-ausf-configmap
          items:
          - key: ausfcfg.yaml
            path: ausfcfg.yaml

部署free5gc 核心網路

依照編號依序部署

切namespace方便查看結果

remote cluster也要做同樣的操做

kubectl create ns 5gc
kubectl label namespace 5gc istio-injection=enabled

依序部署

kubectl apply -f 01-free5gc-mongodb.yaml -n 5gc
kubectl apply -f 02-free5gc-upf.yaml -n 5gc

將nrf部署在remote cluster

# cluster primary部署service即可
kubectl apply -f 03-free5gc-nrf.yaml -l service=free5gc-nrf --context=$CTX_CLUSTER1
# cluster remote部署service、nrf、nrf configmap
kubectl apply -f 03-free5gc-nrf.yaml -l service=free5gc-nrf --context=$CTX_CLUSTER2
kubectl apply -f 03-free5gc-nrf.yaml -l configmap=free5gc-nrf --context=$CTX_CLUSTER2
kubectl apply -f 03-free5gc-nrf.yaml -l version=v2 --context=$CTX_CLUSTER2
kubectl apply -f 04-free5gc-amf.yaml -n 5gc
kubectl apply -f 05-free5gc-smf.yaml -n 5gc
kubectl apply -f 06-free5gc-udr.yaml -n 5gc
kubectl apply -f 07-free5gc-pcf.yaml -n 5gc
kubectl apply -f 08-free5gc-udm.yaml -n 5gc
kubectl apply -f 09-free5gc-nssf.yaml -n 5gc
kubectl apply -f 10-free5gc-ausf.yaml -n 5gc

驗證跨叢集流量

查看nf有無註冊成功

kubectl logs free5gc-nrf-deployment-v2-f557cf589-mh9s2 -n 5gc --context=$CTX_CLUSTER2


上一篇
18-Istio 多叢集容錯(2)-金絲雀部署
下一篇
20-Istio free5gc 核心網路跨叢集容錯
系列文
5哥窟-5G與肝臟買賣30
圖片
  直播研討會
圖片
{{ item.channelVendor }} {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

尚未有邦友留言

立即登入留言